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ABSTRACT 


In an environment of scarce resources and rising federal deficits the people not 
only expect, but demand greater accountability for the spending of public funds. This 
demand has created a trend in the public sector, not only in the United States, but 
worldwide as well, towards the importation of private sector business practices to 
improve accountability-oriented analysis. One example is increased emphasis on return 


on investment (ROJ) analysis in public sector organizations. 


Development and application of ROI analysis is challenging in the public sector 
since most government organizations do not generate profit necessary for calculation of 
ROI in the manner in which it is done in the private sector. This thesis develops the 
methodology necessary for use of ROI analysis in the public sector. ROI methodology is 
applied for test evaluation with the Space and Naval Warfare Systems Command 
(SPAWAR) in San Diego. The test demonstrates that ROI can be applied successfully to 
assess the relative efficiency of value-added work and to improve the process of choosing 
between investment alternatives. Properly designed ROI analysis reveals how and for 
what goods and services money is spent and provides a means for comparing the value 


derived from investment and work performed. 
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I. INTRODUCTION 


A. PURPOSE 


This thesis identifies and assesses several previous efforts to use return on 
investment (ROI) criteria as benchmarks in the public sector. Lessons learned from this 
review have been applied in the development of a notional ROI (NROI) formula 
identified for applicability with the Program Executive Office (PEO) Command, Control, 
Communication, Computers, and Intelligence (C4I) Shipbuilding Coordinator (SC) at 
Space and Naval Warfare Systems Command (SPAWAR). Specifically, the data used in 
the NROI formula apply to the design, procurement, testing, installation, integration, 
personnel training, and certification of the radio control suite (RCS) equipment and 
associated gear for the first ship of T-AKE new construction. The same formula was 
applied for comparison purposes with a similar effort on the LPD-19, third ship of the 
LPD-17 class. This thesis provides a detailed analysis of the strengths and weaknesses of 
applying ROI in the public sector, with specific emphasis on original work performed 


cooperatively with PEO C4I SC. 


B. BACKGROUND 


The Navy and Department of Defense (DoD) are attempting to define and apply 
improved methods for evaluating ROI in various ways, under pressure from Congress, 
the President's Office of Management and Budget and the scarcity of resources available 
for defense (McCaffery and Jones, 2004: 1-20). ROI is one of the key methods used to 
quantify the level of success achieved or achievable in a business endeavor. The concept 
of ROI is used throughout private industry not only to determine past results, but also to 
evaluate the current situation and as a decision making tool for the future. The 
advantages of ROI are clear in that it provides the flexibility to anticipate output changes 
in advance. This benefit results in the ability to not only preview the future in a real 
world sense, but also to modify the inputs to the numerator and denominator of the 


equation to model potential courses of action for the enterprise. 


Although a useful concept, ROI does not easily transition for use in the public 
sector. Unlike private enterprise, the public sector has no “profit” or “total sales” to use 
in the equation. The increasing need for some method to quantify ROI in the public 
arena has led to multiple attempts from a diverse group of public enterprises with varying 
results. The Australian government placed increased emphasis on what they termed the 
“value added” approach in an effort to determine the output they were receiving as a 
result of budgetary expenditures. The Royal New Zealand Navy desired a determination 
of ROI for the implementation of a retention bonus plan used to control the attrition 
problem that was being experienced with marine engineers. Both of these results were 
somewhat mixed with valuable lessons learned. The United States Postal Service (USPS) 
met with a greater level of success in their effort, due largely to the fact that they are run 
much more like a private enterprise. Although not seeking to be “profitable,” the USPS 
does generate revenue which can be used in the numerator of the formula which when 
divided by the USPS asset base in the denominator results in a fairly conventional ROI. 
Finally, the US Navy Dental community effort was much more ambitious in that it 
attempted to convert non cash outputs into cash equivalents in order to closely adhere to 
the traditional ROI formula. The resulting Navy Dental ROI was dogged by the 
questionable accuracy of some inputs, but the overall approach remained fundamentally 


sound. 


The previous efforts focused on in this study have helped define the need for a 
new method to determine ROI in public sector enterprises. The intent of this thesis is to 
review the previous efforts and to develop a new approach for attaining this important 
goal as well as to provide a workable response to the Chief of Naval Operation’s (CNO) 
call for an innovative method of linking investments with measurable outputs. In 
November of 2003, in an Echelon II visit feedback memo, the CNO Admiral Vern Clark 
outlined the need to develop a workable ROI formula: 

This is an area where we are struggling the most. We need to know that 

we are making the right type and level of investment. We have made 

some progress in understanding what we are investing in and have even 

made progress in understanding the output of our processes. However, we 

are struggling to link the two (investment to output). We need to model 

how increases or decreases in investments (people, dollars, and 


technology) will change the output. 
2 


It is with the CNO’s request in mind that the PEO C4I SC T-AKE ROI project 
was initiated. The design, installation and integration of the T-AKE class command and 
control suite equipment -- specifically the Radio Control Suite (RCS) -- including 
ancillary equipment costs is used as the test platform to apply what we term the NROI 
method specifically applicable for use in the public sector. Our definition of NROI is 
based on the asset turnover ratio and return on asset approach as they are referred to in 


private sector accounting and finance. 


C. RESEARCH OBJECTIVES 


The immediate objective of this effort is to develop a workable NROI formula for 
use within the SPAWAR organization, specifically PEO C4I SC. Ina more broad sense, 
however, the underlying objective is to validate an approach to developing NROI 
formulae for use in the public sector across a wide variety of organizations. The 
sequence of development of the NROI formula is reported in this thesis to provide the 


reader with a road map to follow, including valuable examples and lessons learned. 


D. RESEARCH METHODOLOGY 


The methodological approach used in this thesis evolved as the research project 
and process unfolded over a period of approximately ten months. The initial phase 
included extensive (literally worldwide) research of past efforts at determining ROI in the 
public sector. After deriving a number of lessons learned from these past efforts, focus 
shifted to a determination of the criteria that had to be followed for successful NROI 
development. This step was followed by a series of assessments of the NROI formula 
under development for the T-AKE project, including the necessary reduction of scope to 
a focus on solely the RCS. The ensuing stages of formula development and evolution 
incorporated elements from the research on what others had attempted (in the private and 
public sectors) and what was necessary for design and use with the specific units of 
analysis (Navy ships) under study. Finally, the NROI formula was used to compare 


return on investment in the RCS for the first T-AKE class ship in production at different 


points in time, and the RCS for both T-AKE 1 and LPD-19. More detailed articulation of 
the development of the NROI methodology follows. 


Il. TRADITIONAL VS. NOTIONAL ROI 


A. TRADITIONAL ROI 
1. How to Determine Traditional ROI 


ROI has traditionally been measured in the private sector to quantify an 
organization’s past, present, and potential future performance. There are several methods 
by which an organization can determine its ROI. Most compare the net financial output 
of a company, or profit, to the financial input. One of the most common methods is to 
compute a percentage return on a company’s assets. An organization can determine how 
efficiently it has used its assets by comparing a period’s operating income to the total 
amount the company has invested in the assets that produced that income. ROI is 
traditionally calculated as follows (Garrison and Noreen, 2003: 542): 


Net Operating Income 
Return on Investment = P g 


Average Operating Assets 

Net operating income is the difference between revenue and expenses, usually 
before taxes and interest. An average asset base is normally used since the amount of 
assets in use may have changed during the period of measurement. Regardless of the 
exact method of measurement, a higher return indicates a more proficient use of 
organizational assets and ultimately a higher return for its shareholders. 


Ze Practical Applications of Traditional ROI 


ROI calculations also may be used to determine the potential reward of a single 
investment decision or to assist in choosing between multiple investment options. For a 
single investment decision, forecasted streams of revenue are estimated and compared to 
the expected capital investment and operating costs. These are compared over the life of 
the proposed project and used to determine an internal rate of return (IRR), actually a 
forecasted ROI. The IRR is then compared to a firm’s cost of capital for a single 
investment decision. It can also be compared to the IRR forecasted from other 
investment decisions in the case of multiple options to assist in choosing between them. 
With reasonable forecasting accuracy, this becomes an effective tool used in the private 


sector for deciding between capital venture decisions. 


3. Traditional ROI Example 


There are two frequently used methods to determine a corporation’s ROI. 
Consider an investor deciding whether or not to make an investment in PepsiCo in 2002. 
One method for estimating PepsiCo’s future performance is to look at its previous year’s 
use of assets. This is commonly referred to as an organization’s return on assets, or 
ROA. This was the method discussed in the previous section. Selected data for PepsiCo 


taken from 2001 (Brealey, Marcus, and Myers, 2004: 450-51) are presented in Table 1. 














Table 1. Selected Financial Data for PepsiCo: 2001 (millions of dollars) 
Earnings Before Interest and Taxes $4,181 
Net Income (after interest and taxes) 2,662 
Total Assets 21,695 
Total Shareholder Equity 8,648 














To determine PepsiCo’s ROA for 2001, their earnings before interest and taxes 
are taken from the income statement and must be divided by their total assets from the 
balance sheet. The result is then multiplied by one hundred to produce a percentage 


ROA. Thus, for PepsiCo in 2001; 


$ 4.2B 
$21.7B 





ROA (%) = X100=19.3% 


This indicates that every dollar invested in assets at PepsiCo yielded 19.3 cents of return 
in 2001. 


Another more relevant method to the investor would be to determine PepsiCo’s 
return on equity, or ROE for 2001. This explicitly gives the return on investor equity in 
PepsiCo. ROE is determined by dividing the net income (income after interest and taxes) 
by the corporation’s total shareholder equity. The result is then multiplied by one hundred 
to produce a percentage ROE. Thus for PepsiCo in 2001; 


ROE (%) == X100 =31.4% 


This indicates that every dollar invested in PepsiCo by investors yielded 31.4 cents of 
return to the shareholders in 2001. 


It is important to keep in mind that taken on its own the ROI is of limited value. 
In this example it would be wise to compare the ROA and ROE for PepsiCo to prior 
years or with other companies in the same business during the same year. This 
comparability is very helpful in determining if the ROI is superior, average, or mediocre. 
Careful evaluation of the inputs to the ROI formula can uncover what may be the root of 
the success or problem. 


4. Difficulties Utilizing Traditional ROI in the Public Sector 


Public sector ROI calculations are considerably more problematic to utilize than 
in private industry. The traditional method of determining investment returns in the 
private sector is not directly compatible with many public sector organizations. Consider 
how a public sector organization would determine its financial output. Many public 
sector organizations do not produce revenues or generate profits as outputs. Therefore, 
their outputs are difficult to quantify in dollars. Instead, they provide a service or 
capability to the public. Oftentimes this service or capability is unique to the public 
sector and is not produced by the private sector. This increases the difficulty when trying 
to value these unique services or capabilities. For example, how much value is added to 
the respective service when another tank or fighter jet is produced? We certainly know 
what they cost. However, it is difficult to quantify their value added to the Army or Air 
Force. Placing dollar values on these items is complex since similar items are not valued 
in the private sector. The value added to the services from these items cannot be easily, if 


at all, measured in dollars. This makes the use of traditional ROI criteria impossible. 


Some public sector organizations could be measured by the equivalent value of 
the service or capability provided in the private sector. For example, a comparison could 
be made between the USPS and the United Parcel Service. Perhaps a cost comparison for 
compatible services between private and public sector organizations could be used to 
measure performance. However, many public sector companies do not have comparable 
organizations in the private sector. For example, consider the DoD. The DoD provides 
defensive and offensive capability for the United States. This capability cannot be 
measured against the private sector due to the uniqueness of the services it provides. 
Therefore, to facilitate a ROI metric for many public sector organizations, a different 


approach needs to be used. 


B. NOTIONAL ROI 
1. Exploring ROI in the Public Sector 


ROI measurements are under exploration in the public sector for three primary 
reasons. First, as with any private sector corporation, there are significantly more 
investment opportunities than public funds available. There is intense competitive 
pressure between organizations to continually prove their need for additional or even 
continued program funding. Deciding between these alternatives is oftentimes subjective 
in nature since objective data is not available. Realistically, some public organizations 
will be funded regardless of their ROI. However, ROI measurements could provide one 
metric to objectively decide between investment alternatives in public programs. They 
could also be used by organizations to show their value added to the public, and 


consequently provide support for their continued funding. 


Second, increased public spending and rising budget deficits have considerably 
raised the public’s concern for the way the public sector spends its money. There has 
been a notable increase in the required accountability of the public sector to the 
taxpayers. Evidence of this is the Government Performance and Results Act of 1993. 
The general purpose of this legislation is to establish metrics within the United States 
government to hold organizations accountable (Phillips and Phillips, 2002: 4). ROI 


measurements are one way this accountability requirement to taxpayers can be satisfied. 


Finally, there has been a long and continuing trend within the public sector to 
import successful business practices from the private sector. This is no surprise since 
many of the public sector’s leaders had previous careers in the private sector. In fact, 
three of the public sector’s most senior leaders, the current United States President, Vice 
President, and the Secretary of Defense, are all previous chief executive officers from the 
business community. In addition, the private sector is generally viewed by the American 
public and many academics as more efficient than the public sector. Unless a private 
organization produces a unique product or service, efficiency is essential for competitive 
survival. These same pressures are not present in the public sector to the same degree 
and facilitate the increased efficiency in the private sector. Consequently, the public 


sector oftentimes looks to mimic the more efficient practices in the private sector. ROI 


measurement is just one of many business practices within the private sector that is under 
experimentation in the public sector. 


2. Non-Monetized Notional ROI 


The inability to directly apply ROI techniques to many organizations in the public 
sector indicates that a different approach is required. One such approach includes the use 
of cost effectiveness analysis to provide a useful framework with which to assign weights 
to the numerator variables. In Cost Benefit Analysis: Concepts and Practice, the authors 
address this issue: 

If the analyst is unable...to monetize the major benefit, then cost- 

effectiveness analysis may be appropriate. Because not all of the impacts 

can be monetized, it is not possible to estimate net benefits. The analyst 


can, however, construct a ratio involving the quantitative, but non- 
monetized, benefit and total dollar costs (Boardman, et al., 1996: 36). 


This was the initial approach used for analyzing the PEO C4I SC T-AKE project. The 
resulting formula would produce a non-monetary output considered to be the NROI. As 
changes to the numerator variables or the denominator are considered, a new NROI can 


be generated to determine the potential impact of the changes. 


In order for the NROI formula to be of credible value, weights must be assigned 
to each of the numerator variables. Weights are indicative of how the decision makers 
prefer to balance the impact of the attributes. This step is extremely important since the 
weight distribution has a tremendous impact on the output. Determination of weights can 
be an objective result of models and data analysis, a subjective result of discussion by the 
decision makers, or a combination of both. There are four common methods for 
determining weights: equal weighting, rank reciprocal, pair-wise comparison, and direct 
assessment (McNab, 2004). Table 2 presents a comparison of the various methods using 


the six categories from the initial NROI formula for the T-AKE project as an example. 


Table 2. Weighting Method Comparison 





Weighting Methods 
Effectiveness Equal Rank Pair-wise Direct 
Weighting Reciprocal Comparison Assessment 

Cp . 

Capability .167 1/1360/147 939/65 10 
Cy 

Cost 167 1/2930/147  11311/65 25 
Sx 

Schedule 167 11320/147 1717/65 gg 
Q; es 

Quay 167 4/4-315/147 44/65 30 
E, Effectiveness 

& Efficiency 167 = 1/5312/147 1313/65 15 
Ip Intellectual 

: 167 1/6310/147) = 11311/65 15 
Properties 


The equal weighting method simply assigns equivalent weights to all of the 
variables. The rank reciprocal method has four steps. First, each variable is ranked in 
order of relative importance (1 — 6 in our example). Next the reciprocal of the ranks is 
taken (1/1, 1/2, 1/3, etc...). The resulting fractions are then added together using a 
common denominator to create a new base (60/60 + 30/60 + 20/60 + 15/60 + 12/60 + 
10/60 = 147/60). Finally, the original reciprocals for each variable are divided by the 
new base (147/60) with the resulting distribution being used for weighting (see Table 2). 
The equal weighting and rank reciprocal methods generally do not provide a high enough 
level of subjective scrutiny to be of value in a detailed project. With the pair-wise 
comparison method, the decision makers are provided a specific number of points to be 
distributed as they see fit between the variables (65 points for the example in Table 2). 
After discussion, each variable is assigned a numerical value. The sum of the values is 
then used as the denominator for the variable weighting, with the numerator being the 
assigned numerical value. Like the previous two methods; pair-wise also fails to provide 
enough ability to fine-tune the weighting distribution for a detailed project. The direct 
assessment method uses deductive reasoning to determine and assign weights to each 
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variable. Although this method is purely subjective, it is less random and can easily be 
modified as necessary. The subjective nature of direct assessment can be alleviated to 
some extent by using a number of technical experts to develop the weighting values to be 


used in the formula. 


Once weightings have been assigned, the non-monetized value of the numerator 
variables can be determined. After adding the variables together, the resulting numerator 
value is divided by the asset base to provide an NROI output. The validity of this NROI 
on its own is minimal. Trend analysis is required, using subsequent alterations to the 
numerator variables for the first ship in the class as it is completed and compared with 
independent data from the second ship in the class as it progresses. Essentially, the first 
NROI developed sets a baseline that is used to compare with subsequent outputs for the 
same ship. The trend data from the first ship can then be analyzed to determine if 
priorities can be adjusted for the second ship in order to improve the output. 
Additionally, comparisons can be made between ship classes provided the category 
modifiers are the same and the scope of the project is similar; as is the case with the RCS. 


3. Monetized Notional ROI 


While it is important to provide weights to the categories used in the numerator of 
the equation, it is even more important if at all possible to use monetary values for the 
NROI formula. In The Bottomline on ROT, while discussing the determination of ROI in 
the public sector, Patricia Phillips states that “converting data to monetary benefits is 
critical...the process is challenging, particularly with soft data, but can be methodically 
accomplished” (2002: 73). For the formula to be most applicable for the purpose of 
comparing the past, present, and future NROI of a number of projects, it was determined 
that the effort must be expended to convert the data to monetary values. This would also 
serve to provide an NROI as near to traditional as possible for a public sector 


organization. 


As formula development progressed it became clear that the final formula would 
be a combination of both monetized and non-monetized approaches. The cost data would 
be actual monetary values broken down into a number of categories. Each category 


would then be modified by a non-monetized value added factor (VAF) determined by a 
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number of system experts using the direct assessment approach. The resulting formula 
would incorporate the essential cost elements while also attempting to capture the 


intangible benefits provided by the public sector entity in the absence of profit. 
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Hil. PREVIOUS EFFORTS AT DEVELOPING ROI IN THE 
PUBLIC SECTOR 


A. AUSTRALIA 
1. Scope and Reasoning behind the Effort 


In an attempt to justify government acquisitions, Australia’s government 
continues to focus on improving its own ability to develop and implement ROI criteria. 
In a society vigorously competing for scarce public resources, receiving the best value for 
money spent has become central to government policy. The Australian Commonwealth 
demands this accountability. In response, Australia’s government has taken strides to 
emphasize the development of ROI criteria to make acquisition decisions throughout its 
governmental departments. However, we were unable to find a specific example where 
ROI criteria were successfully developed and implemented by a governmental 
organization. Dr. Allen Hawke, Australia’s previous Secretary of Defense, acknowledges 
the public’s frustration with their lack of success to date. According to Dr Hawke’s 
address in February of 2000, “there is a widespread dissatisfaction with Defence’s 
Performance (regarding Australia’s Defense Organization use of funds)...In essence we 


have a credibility problem.” 


Australia’s Department of Finance and Administration (ADOFA) is responsible 
for providing direction to Australia’s ministries in making procurement decisions. 
Instead of simply choosing the lowest cost alternative, ADOFA emphasizes the 
“achievement of value for money” (2003). Among other things, this method weighs the 
ability of the alternatives to meet the stated objectives, the reliability and reputation of the 
contractor, and the whole of life costs instead of just the initial procurement cost. Instead 
of providing structured guidance to determine ROI, ADOFA provides a substantial list of 
things to consider and leaves it to the particular agency to identify and weigh those things 
that apply. Due to the unique benefits of each procurement decision, this general 
approach may be appropriate. However, recent comments from Australia’s Defense 
Procurement Review indicate a lack of success thus far within the acquisition 


community. The review concludes with the following comment; 
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Our review of the acquisition process has led us to conclude that there is 
no single cause of the failures that have become apparent in the 
development of capability and the acquisition and support of defence 
equipment (Australia Dept. of Defence, 2003: 47). 


2. Lessons Learned 


The Australian government does acknowledge the need to consider ROI when 
making procurement decisions. However, by merely emphasizing value for money in 
broad terms, they are not actually implementing quantitative ROI criteria within their 
government. They do highlight the need to consider many important factors other than 
costs for procurement decisions such as quality and contractor performance. Yet, they do 
not provide a universal method for considering the weighting of these factors so that 
decisions can be consistently made the same way across the different ministries. Perhaps 
this inconsistency is one of the reasons for their continued lack of success within the 


Department of Defence acquisition community. 


B. NEW ZEALAND 
1. Scope and Reasoning behind the Effort 


The following ROI case concerning the Royal New Zealand Navy (RNZN) is 
summarized from a case study authored by Beryl Ann Oldham, Paul Toulson, Brenda 
Sayers, and Graham Hart (2002). The RNZN had encountered considerable difficulty 
retaining their marine engineers (ME) in the mid 1990s due to high attrition rates. The 
ME community is responsible for many of the complex systems aboard the RNZN’s fleet 
ships including operation and maintenance of diesel engines, gas turbines, electrical 
generators, and air conditioning and refrigeration plants (Royal New Zealand Navy, 
2004). The attrition problem was so significant that the ability of the RNZN to maintain 
an acceptable operations tempo was threatened. Several suggestions were made in an 
effort to reduce the ME attrition rate. These measures included improved ME career 
management initiatives, better management of leave and maintenance periods, improved 
pay, compensation time for working weekends, and the more controversial Marine 
Engineer Retention Bonus Scheme (MERBS). It was believed that implementation of an 


immediate retention bonus was imperative to control the attrition problem in the short run 
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since the other proposed initiatives would be slower to take effect (Oldham, et al., 2002: 


132). 


The MERBS was an expensive human resource endeavor for the RNZN. MERBS 
costs included both administrative program set up costs and the retention payments to 
personnel themselves. These overall costs were estimated at almost five million 
Australian dollars (144). However, the MERBS was considered a successful initiative 
since it did reduce attrition rate for the MEs to an acceptable level. Unfortunately, it was 
difficult to determine just how successful the MERBS was, especially considering that 
other retention initiatives were occurring simultaneously. Consequently, an ROI study 
was conducted to determine the isolated effect of the MERBS on ME retention in the 
RNZN. 


The monetary benefits of any retention program are the avoided expenses for 
replacement and training of new personnel and the separation costs incurred for personnel 
leaving the military. There are also some less tangible benefits including higher 
experience levels, improved morale and increased flexibility. However, in order to 
remain objective, the focus of the study was placed on the monetary benefits achieved by 
the MERBS. There were two approaches taken to isolate the monetary benefits of the 
MERBS on retention. 


The first approach was more subjective in nature and involved the use of a 
questionnaire taken by both the participants in the MERBS and their managers. The 
questions were tailored to evaluate the effectiveness of the MERBS and its isolated 
impact on the ME participants to stay in the RNZN. It was ultimately determined, based 
on these questionnaires, that forty one percent of the participants’ decisions to stay in the 
RNZN were influenced by the MERBS. These forty one percent were then asked to rate 
the accuracy of their answer regarding the influence of the bonus payments on their 
decision to stay. The reply indicated they were ninety-three percent confident in the 
accuracy of their answer regarding the MERBS influence on their decision to stay (138). 
The actual monetary benefit was determined by multiplying the participants’ impact 


estimation of the retention payments by the estimated savings of retention for each of the 
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170 personnel participating in the MERBS. The estimated savings per participant and the 
detailed calculation of the ROI from this approach is shown in Table 3. 


Table 3. Determining ROI with the Participant Impact Estimation (Oldham, et 
al., 2002: 140, 145) 





Number of participants in the MERBS at the end of three year period = 170 





Estimated separation cost per ME leaving the service = $4,260 





Average replacement cost per ME leaving the service = $105,133 





Percentage of decision to stay influenced by retention payments = 41% 





Confidence in decision to stay influenced by retention payments percentage = 93% 





Participants estimation of retention payment’s impact = 0.41 x 0.93 = 0.38 





Monetary Benefits = ($4,260 + $105,133) x 170 x 0.38 = $7,066,789 








ROI from participant impact estimation = Net Benefits/Program Costs x 100 
= ($7,066,789 - $4,926,504)/($4,926,504) x 100 
= 43 % 





The second approach involved a more objective approach using retention trend 
data. Predicted turnover of ME personnel without the retention payments was estimated 
based on historical trends in both ME and non-ME personnel prior to the MERBS period. 
Based on previous trends of non-ME and ME personnel before the MERBS initiative, 
ME turnover averaged 5.5 percent higher than non-ME personnel (Oldham, et al., 2002: 
141). These data were used to determine an expected ME turnover without the retention 
payments. This was compared to the actual turnover during the MERBS period to 
determine an actual number of ME participants that were retained as a result of the bonus 
payments. Based on this comparison, it was concluded that seventy-three additional 
personnel were retained during the MERBS period than was predicted based on trend 
data without the MERBS initiative. The detailed determination of the ROI based on this 


second approach is shown in Table 4. 
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Table 4. Determining ROI with the Forecasting Method (Oldham, et al., 2002: 
144-45) 





Number of personnel retained attributed to MERBS initiative = 73 





Estimated separation cost per ME leaving the service = $4,260 





Average replacement cost per ME leaving the service = $105,133 





Monetary Benefits = ($105,133 + $4,260) x 73 personnel = $7,985,689 








ROI from Forecasting Method = Net Benefits/Program Costs x 100 
= ($7,985,689 - $4,926,504)/ ($4,926,504) x 100 
= 62% 





2. Lessons Learned 


It is interesting to note the similarities and differences between the two 
approaches. Both approaches determine program costs the same way. Even though the 
methods of determining monetary benefits are very different, the results are surprisingly 
similar. The participant impact estimation yields a monetary benefit of approximately 
seven million dollars while the forecasting method yields a monetary benefit of 
approximately eight million dollars. However, these numbers do yield significantly 


different ROI for the MERBS initiative. 


The question then becomes which approach is more valid? Both approaches are 
logical and defendable. The forecasting approach is more objective since it is based 
entirely on data and trend analysis. However, the shortcoming is that it does not entirely 
isolate the effect of the MERBS initiative from the other retention initiatives occurring 
simultaneously. It is plausible that most of the seventy-three additional personnel 
retained were a result of the MERBS, but it is certainly possible that other proposed 
initiatives played a factor. The participant impact estimation approach is clearly more 
subjective since it is based on responses from a questionnaire. Conversely, it does better 
address the isolated effect of the MERBS initiative through the inclusion of specific 
questions in the questionnaire. Unfortunately, ROI methodologies based on non- 
traditional methods cannot always be purely objective. This is what makes it so 
challenging in the public sector. The objective is to develop a credible methodology, 
which is what was accomplished here. Therefore, both methodologies are valid as long 


as their shortcomings are kept in mind. This is true of any mathematical formula. 
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A lesson learned discussed in the case relates to data collection. Ideally, the 
decision to perform an ROI determination is made prior to program implementation. 
This way data required can be determined and recorded while the program is taking 
place. For this case, the decision to determine the ROI for the MERBS was not made 
until after the program was implemented and well under way. Therefore, many data 
collection opportunities were missed. The recommendation of the study is to consider 
ROI evaluations and associated data collection needs during the development phase of 


program initiatives if possible (Oldham, et al., 2002: 146). 


Another lesson learned from the case was the extent taken to keep the ROI 
methodology simple. They could have attempted to determine monetary values for more 
complex but less tangible benefits. For example, they could have attempted to determine 
the monetary benefit of the increased operations tempo or increased experience levels 
available because of the higher retention rate. However, it is easy to imagine that this 
would involve potentially long and complex mathematical formulae and additional 
subjectivity. By avoiding these attempts, the ROI methodology is easier to understand 


and more credible. 


C. UNITED STATES POSTAL SERVICE 
1. Scope and Reasoning behind the Effort 


The USPS used an Economic Value Added (EVA) program to determine their 
ROI from 1996-2002. EVA was calculated by determining the net operating income and 
subtracting a fee proportional to the cost of the assets used to produce that income 
(USPS, 1997). This difference represented a positive net cash flow that added financial 
value to the post office. A higher EVA indicated a more efficient use of assets. 
Consequently, senior post office executives were rewarded for performance at the USPS 
based on this figure. This provided financial incentive for post office employees to seek 
out new and better ways to improve efficiency within the organization. This program 
was credited with contributing to the $3.5 billion in net income earned by the USPS from 
1996-2000 (USPS, 2002). However, amid strong controversy relating to the calculation 


of EVA program incentive bonuses, the effort was abandoned in 2002. 
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The USPS 2003 Annual Report cites the continued use of ROI criteria for capital 
venture decision-making. Unlike many public sector organizations, the USPS is one of 
the few public sector companies that generate revenue. This facilitates using the 
traditional method of calculating ROI. The USPS continues to invest in automation 
equipment to reduce personnel work hours in mail processing and delivery (USPS, 2003: 
25). The cost savings realized from automation is then compared to the cost of acquiring 
the required equipment for making procurement decisions. The USPS also uses a Cash 
Flow/Capital Expenditure (CAPEX) ratio as a benchmark for assisting in making capital 
purchase decisions (USPS, 2003: 24). The additional yearly cash flow to operations is 
compared to the yearly cash outlays to support the project. This helps to determine the 
attractiveness of the proposed project and the required need to borrow funds to support it. 


2. Lessons Learned 


The basis for determining economic value added at the USPS had significant 
flaws. The idea of subtracting additional costs from increased benefits to determine value 
added from a given investment is sound. However, the application of the EVA Variable 
Pay Program was inconsistent when overall USPS performance is considered. For 
example, the USPS lost $199 million in fiscal year 2000 but still paid out over $280 
million in performance bonuses (Lexington Institute, 2001). It is counterintuitive for an 
organization to lose money in a given fiscal year and still pay out such a significant sum 
of money in performance bonuses. In fact, the reason the USPS lost money that year was 
due to the bonus payouts. Inconsistent ROI measurements such as EVA are not credible 
if they indicate positive results when other metrics such as negative net income indicate 


to the contrary. 


The USPS uses simple and intuitive methods for determining ROI for evaluating 
investment decisions. The use of cost savings as a basis for determining ROI is a 
commonly used method for public sector organizations. This is because many 
procurement decisions made by public sector organizations involve investments that will 
ultimately improve efficiency. If these efficiencies are able to be quantified, they can be 
used as a basis for comparison to the required capital expenditure to determine an ROI. 
The CAPEX ratio used by the USPS to evaluate capital purchase decisions is also an 


intuitive way to determine an ROI. Comparing cash flows to required capital 
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expenditures is very similar to a net present value calculation commonly used in the 
private sector to evaluate investment alternatives. It is important to emphasize that the 
USPS does generate annual revenues which lends itself to the use of traditional ROI 


metrics. This is uncommon among most other public sector organizations. 


D. NAVY DENTAL CORPS 
1. Scope and Reasoning behind the Effort 


In response to the CNO’s call for better decision making tools, the Navy Dental 
Corps (NDC) has developed a simple metric to determine ROI at the branch clinic level. 
Captain York, the navy representative at the Tri-Service Center for Oral Health Studies 
(TSCOHS), spearheaded the effort to determine a practical method for defining NDC’s 
return for investment dollars. While not using the traditional method of ROI that 
compares earnings to assets, this effort provides an easily understood metric to quantify 


performance at the branch clinic level. 


NDC’s ROI formula compares a branch clinic’s quarterly output, defined as 
Dental Weighted Values (DWVs), to its required investment in funding dollars and 
military labor. The formula is as follows (Mitton, 2004): 





(Otrly DWVs x 100)—((Branch APF + Annual Branch Milab) x 0.25) oe 


ROI(%) = 
%) (Branch APF + Annual BranchMilab) x 0.25 


00 
DWV (3) — Dental Weighted Values 


Branch APF ($) — Branch Clinic Annual Planned Funding 


Annual Branch Milab ($) — Branch Clinic Military Labor 


Branch APF is the operation and maintenance funding allocated to the clinic. Both 
Branch APF and Annual Branch Milab are converted to quarterly values to determine 
quarterly ROI. DWVs and Annual Branch Milab are determined through separate data 


collecting programs described next. 
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DWVs are determined by input from the branch clinic into a program known as 
DENCAS. The clinic enters the different procedures performed on a given day using 
American Dental Association (ADA) procedural codes, known as Common Dental 
Terminology (CDT) codes. CDT codes are converted into DWVs that are essentially 
equal to one hundred dollars worth of dental services. This result is multiplied by one 


hundred to convert the DWVs directly into dollars for use in the ROI formula. 


Annual branch clinic labor is determined by the collection of data into the 
Medical Expense and Performance Reporting System (MEPRS). Branch clinic 
employees specifically document their hours worked performing a variety of individual 
tasks on MEPRS sheets. Different tasks such as various medical duties, training, and 
even leave/liberty times are documented. These data are correlated at the comptroller 
level to determine military labor hours and is converted into a dollar figure based on the 
rank and rate of the military employees working at the clinic. 


2: Lessons Learned 


The Navy Dental ROI formula discussed above is no longer used for three 
reasons. First, there is significant skepticism regarding the quality of the data being 
tracked for use in the calculation of ROI. Specifically mentioned was the inaccurate data 
collected by MEPRS. Many dental employees failed to log their hours on a daily basis; 
instead, they would record their hours on a weekly or monthly basis. This brings the 
accuracy of the type and number of hours into question due to the delay time in 
recording. Many times employees would wait until the end of the month and simply log 
eight hours of work arbitrarily for each day. Commander Mitton, from the Navy Bureau 
of Medicine and Surgery in Washington DC, referred to this popular method as “logging 
straight eights” (2004). 


It is also difficult to use this formula for comparison. Different branch dental 
clinics may be responsible for different operating costs. For example, some of the clinics 
are responsible for paying their rent and utilities while other clinics are provided with 
these resources free of charge directly by the base command. This directly affects the 
amount of Branch APF the clinic would receive and, consequently, affected the results of 


the ROI formula. 
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Finally, the Navy Dental ROI formula does not include many of the cost elements 
required to staff and operate a branch dental facility. For example, large expenses such 
as the cost of training Navy dentists and dental technicians are not included. Other large 
costs such as accession bonuses for dentists and depreciation expenses for major 
equipment are also not included. Therefore, ROI for the Navy branch dental clinics 


needed to be more adequately defined. 


As a consequence, Captain York developed a more robust formula to be used in 
calculating ROI for Navy Dental clinics. Although similar to the previously discussed 
formula, it also includes many of the lacking cost elements. See Table 5 on the following 
page for a sample calculation using Pearl Harbor for FY 2000. ROI is calculated as 
shown below (York, 2004): 


ProductionValue—Cost of Production 


ROI (%)= 
o) Total Cost of Production 


x 100 





Production Value is determined similar to the DWVs calculated in the original 
formula. The Cost of Production, however, includes significantly more cost elements. 
For example, note the system costs listed in Table 5. These are the allocated training 
costs to the Pearl Harbor dental clinic from the dental training pipeline. These were not 
included in the original formula but are real costs burdened by the NDC and should be 
included. These system costs are divided between all the clinics proportionately based on 


the number of dental technicians and dental officers employed at the clinic. 
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Table 5. Raw Data: Pearl Harbor Branch Dental Clinic FY2000 (York, 2004) 


FY 00 COSTS (EXCLUDING DENTAL COST AND PRODUCTION ABOARD SHIPS) 


Command/Branch: PEARL HARBOR 


Number of command/BDC Dental Officers | 28 ~—_| (to determine system costs) 
Number of military dental techs. (incl. ships) 2807 


NDCs NHs SYSTEM TOTAL 
Military Pay (Claimancy 18) $ 5,708,382 $ 86,309 $ 5,794,691 
NDRI Military Pay $ 18,694 $ 18,694 
Dental Bonus (above composite) $ 368,186 $ 368,186 
Dental Accession Bonus $ 9,146 $ 9,146 
Civilian (GS) $ 449,140 $ 449,140 
Contracts $ 548,086 $ 548,086 
Reserve (composite including travel) $ 529 $ 529 
Dental Scholarships $ 207,520 $ 207,520 
Outservice Training (DO) $ 8,663 §$ 8,663 
Officer Indoctrination School $ 22,041 $ 22,041 
DT "A" School (wio student pay) $ 68,517 $ 68,517 
DT "A" School Student Pay $ 66,237 $ 66,237 
DT "A and C" School Staff Pay $ 20,562 
DT "C" School $ 57,157 $ 57,157 
RDH Training Program $ 3,080 $ 3,080 
Officer Recruiting $ 5,828 $ 5,828 
Enlisted Recruiting $ 104,807 $ 104,807 
Recruit Training $ 103,770 $ 103,770 
MED-06/OODC $ 1,359 $ 1,359 
NDRI Operating Funds $ 40,757 $ 40,757 
Material (equip & supplies) $ 464,619 $ 464,619 
Equipment Depreciation $ - 
Other Procurements Navy (OPN) $ - 
Reimbursable $ 48,457 $ 48,457 
Free Receipts $ - 
All Other $ 385,049 $ 385,049 
Total Cost of Production $ 7,604,262 $ - §$ 1,192,635 


Production Value minus Cost of Production 


Production Value * $ 9,973,433 $ = $ 9,973,433 


ROI 13.4% 
* Based on the 1995 ADA Survey of Dental Fees (75th percentile), adjusted to 1997 dollars using 
the Consumer Price Index for Dental Services. 
Production data source: Navy Executive Information System (EIS), Dental Weighted Values. 


There is still significant variation between the branch dental clinics using the 
improved ROI formula. Table 6 compares the ROI calculated by the above formula for 
all the navy branch clinics. Note that the ROI varies from approximately negative ten 
percent to over one hundred percent. These variations are not due solely to differences in 
performance levels. For example, NNDC is responsible for the costs of the Naval Dental 
Postgraduate School. Due to this fact, the NNDC ROI includes the impact of manpower 
costs and low productivity of the student-body significantly reducing their ROI. 


23 


Therefore, even though this formula is more robust, one must consider more than just the 


final ROI output to fairly compare commands. 


Table 6. ROI for Navy Branch Dental Clinics 

COMMAND COST PRODUCTION VALUE PROD - COST ROI 
GREAT LAKES $28,476,040 $60,746,220 $32,270,180 113.3% 
OKINAWA $18,156,185 $30,022,556 $11,866,371 65.4% 
MID ATLANTIC $29,003,564 $46,150,165 $17,146,601 59.1% 
PARRIS ISLAND $10,810,353 $15,747,470 $4,937,117 45.7% 
CAMP PENDLETON $19,227,602 $26,027,670 $6,800,068 35.4% 
SOUTHWEST $34,282,076 $45,365,397 $11,083,321 32.3% 
SOUTHEAST $19,366,135 $24,946,372 $5,580,237 28.8% 
GULF COAST $14,541,505 $18,741,273 $4,199,768 28.9% 
CAMP LEJEUNE $17,474,322 $22,031,239 $4,556,917 26.1% 
EUROPE $12,454,115 $15,642,759 $3,188,644 25.6% 
PEARL HARBOR $8,796,897 $9,973,433 $1,176,536 13.4% 
FAR EAST $13,896,531 $15,376,018 $1,479,487 10.6% 
NORTHEAST $11,073,148 $12,213,250 $1,140,102 10.3% 
NORTHWEST $9,987,892 $9,711,583 -$276,309 -2.8% 
NNDC $32,151,242 $29,096,131 -$3,055,111 -9.5% 
ALL NDCs $279,697 ,607 $381 791,536 $102,093,929 36.5% 
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IV. NOTIONAL ROI FORMULA DEVELOPMENT FOR SPAWAR 
PEO C4I AND SPACE NEW SHIP CONSTRUCTION 


A. T-AKE PROJECT 





Figure 1. T-AKE 1: USNS Lewis and Clark (Global Security, 2004) 


1. Ship Class Background Data and Building Plan 


The T-AKE class of ships will provide Naval forces at sea with replenishment 
capability involving ammunition, food, repair parts, consumables, and to a lesser extent 
marine and aviation fuel. The initial plan is for twelve ships to be constructed in three 
distinct sets of four ships each known as flights (Program Executive Office, 2003: iv). 
The role of PEO C4I SC with regard to the T-AKE is to “design, procure, integrate, test, 
deliver, install, train, and logistically support the T-AKE C4I suite of equipment” (iv). 


SPAWAR Systems Center Charleston (SSCC) was tasked with configuring the 
RCS portion of the overall installation. The RCS consists of all shipboard equipment 
used for voice and data communications. In addition to T-AKE, SSCC is also 
responsible for the RCS configuration on the LPD-17, LHD-8, and CVN-68 classes of 
ships. By retaining sole responsibility for the RCS across a number of platforms, SSCC 
translates lessons learned from one class of ships directly onto other classes, limiting the 
duplication of errors prevalent in stove-piped efforts. The assemblage of concentrated 


experience and talent at SSCC provides an environment whereby problems can be 
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worked through quickly and efficiently and most importantly where developed better 


business practices can significantly improve the deliverables. 


Initial engineering efforts to plan the design for the T-AKE RCS began in the 
fourth quarter of 2001. This effort continued into early 2004 with necessary 
modifications made along the way. Concurrently, design of the Test Integration Facility 
(TIF) and procurement of equipment began in the fourth quarter of 2002 and continued 
on into the third quarter of 2004. Transfer of the tested RCS will be conducted in the first 
quarter of 2005. USNS Lewis and Clark, as T-AKE 1 will be called when completed, is 
being built at the NASSCO shipyard in San Diego, CA. She is expected to be 
commissioned in mid 2005. 


Zz Turnkey Approach 


With the conventional shipbuilding approach, all equipment destined for 
installation on the ship had to be identified well in advance, typically 5-9 years with 
regard to RCS equipment (Program Executive Office, 2003: 2). This extensive lead time 
led to many problems such as cost overruns due to technology refresh requirements, 
engineering rework caused by the need to make modifications to support new equipment, 
and in some cases no cost impact but rather an efficiency impact as outdated or even 
obsolete equipment were left on board. A more novel method called the Turnkey 


Approach is now being utilized by SPAWAR. 


With the Turnkey Approach, design deadlines are developed with regard to space 
and power requirements but responsibility for installation rests with PEO C4I SC and not 
with the shipbuilder. In this way the shipbuilder can continue on with ship construction, 
providing the spaces and cable routes as prescribed by PEO C4I SC but leaving the 
installation and integration of the actual equipment up to PEO C4I SC. This allows for a 
dramatic reduction in lead time for the procurement of the equipment from 5-9 years 


down to 2 years (2). 


The key element to successful implementation of the Turnkey Approach is the 
TIF located at SSCC. The TIF is essentially a mock up of the eventual RCS for a 
particular ship. The actual equipment that will be used on the ship is installed and 


integrated in advance at the TIF vice the conventional method onboard ship. This allows 
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for the technicians to work out the problems coincident with any such endeavor. Once 
the TIF RCS is up and running at top efficiency the components are packed and shipped 
to the shipboard site for actual installation. In stark contrast to conventional methods, the 
installation of the RCS systems is not normally done until approximately 140 days prior 


to the builder’s ship trials. (Program Executive Office, 2003: 3) 


B. NROI FORMULA SEQUENCE OF DEVELOPMENT 


The sequence of development for the SPAWAR/PEO C4I SC NROI application 
spanned over ten months of time and involved several face-to-face and video 
teleconference (VTC) meetings with Command personnel. Since SPAWAR is a public 
sector organization, they do not report a net income which would traditionally be used in 
the private sector to determine a corporation’s ROI. Therefore, our task was to develop 
an NROI formula that would credibly measure the PEO C4I SC ROI for the T-AKE 
project. This evolutionary process produced four iterations before deciding on the final 
NROI formula in an effort to fairly represent the added value contributed by 
SPAWAR/PEO C4I SC to the T-AKE project. 


The NROI formula development process had to be approached considering a 
number of criteria in an effort to make it both acceptable and useful. Nine non-mutually 
exclusive criteria applied in development of the NROI method included: simplicity and 
intuitiveness, feasibility of implementation, credibility, theoretical adequacy and utility, 
flexibility, validity, ability to be replicated, comprehensiveness, and consideration of all 
program costs. These must all be taken into account, as per advice provided in Patricia 
Phillips The Bottomline on ROI, and are summarized below as related to development of 


the NROI for PEO C4I SC (2002: 60-62). 


Simplicity and Intuitiveness: NROI formulae should be straightforward and 
easily understood. Complex NROI formulae made up of long mathematical equations 
serve little purpose and can be unnecessarily confusing to those they are intended to 
assist. NROI formulae should be quickly comprehended, thus providing managers a 


useful tool to understand the information provided for their use. 
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Feasibility of Implementation: Implementation of a new performance 
measurement for the organization can be time consuming and costly. For example, the 
requirement of a new data collection method to measure performance might require 
significant managerial resources and dollars to employ. Therefore, consideration must be 
given to the feasibility of the elements that make up the NROI formula in order to 
minimize this burden. In cases where it is possible, utilization of existing data collection 


methods (provided they are relevant) is recommended. 


Credibility: The developed NROI formula must be considered credible by those 
whose efforts it is intended to measure and by the managers reviewing the results. A 
performance measurement not considered credible by those tasked to implement it is 
useless. It will either not be implemented as it was intended or will simply be 
disregarded. Whenever possible, it is imperative to use objective measures to represent 
the outputs that are used to measure performance. In cases where this is not possible, 
steps must be taken to minimize the formula’s subjectivity. Finally, a conservative 
approach must be considered when choosing between formula alternatives in order to 


preserve the credibility of the process. 


Theoretical Adequacy and Utility: The NROI formula developed must be 
consistent with other measurements of ROI. According to Patricia Phillips; “The ROI 
methodology must be theoretically sound and based on generally accepted practices” 
(2002: 61). The numerator must represent the organization’s net benefits and the 
denominator must represent the total costs associated with achievement of those net 


benefits. This enables utility similar to private sector uses of ROI. 


Flexibility: The developed NROI formula must be flexible enough to be used in 
several applications. This includes the assessment of the ROI during various phases of 


the T-AKE project as well as the program’s overall ROI. 


Validity: Validity of the developed NROI formula must be verified through 
actual application to the organization. It should produce output consistent with 
expectations. For example, an increase in non-value added costs should reduce the 


organization’s NROI. 
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Replicable: The formula should allow replication of results by independent 
sources provided the same data are available. Once developed and approved, the formula 
should be able to be applied objectively, even though it may contain some subjective 


elements. 


Comprehensiveness: All relevant components of an organization that add value 
to the final products and services should be represented in the formula if possible. Some 
outputs may be difficult to quantify, and must be noted as intangibles. However, 
maximum inclusion of benefits is necessary for fair valuation of the organization’s 


outputs. 


Consideration of All Program Costs: Fair assessment of the organization’s ROI 
also involves accurately accounting for all the program costs that are incurred in 
producing the output. These include the direct costs of the project as well as a fair share 


of the corporate costs or overhead incurred by the organization. 


The most basic ROI formula is shown below (Phillips, 2002: 19): 


Net Program Benefits 


ROI (%) = x 100 


Program Costs 


This was chosen as the baseline template to be used in the development of the NROI 
formula for PEO C4I SC. Net program benefits are benefits minus the costs of achieving 
those benefits. Program costs will include both direct costs of the program as well as a 
fair share of the corporate costs. An ROI determined directly from this formula yields a 
percentage ROI similar to the ROI formulas used in the private sector. 


1. 1° Iteration 


Dr. Lawrence R. Jones, RADM George F. A. Wagner/SPAWAR/PEO C4I and 
Space Professor of Public Management in the Graduate School of Business and Public 
Policy (finance curriculum) at the Naval Postgraduate School (NPS) in Monterey 
spearheaded a new approach to solving the public sector ROI dilemma. Beginning in late 
2003 Dr. Jones worked closely with (RADM ret.) William Marshall, an independent 


contractor from American Systems Corporation (ASC) serving as a consultant to PEO 
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C4I SC in San Diego. Their goal was to think through the application of ROI to the Navy 
and PEO C4I SC in response to the call for such action by the CNO. A value-added 
approach to determine an NROI was developed by Jones with assistance from Marshall, 
Tom Sommers and other PEO C4I SC staff. After several months of serious dialogue on 
the task at hand, the decision was made by Joe Mayer, head of PEO C4I SC, to move 
forward with a specific project. The platform to be modeled was the C4I suite of a new 
class of supply ship (T-AKE). The role of PEO C4I SC is the installation, integration and 
testing of the suite as well as the personnel training required for the operators. Ironically, 
the first ship and class name is Lewis and Clark, named for the two explorers who led a 


visionary project of exploration. (United States Dept. of Defense, 2003) 


The original NROI formula was developed with the help of a number of 
SPAWAR/PEO CA4I SC representatives, led by Jones, Sommers and Marshall, at their 


semi-annual meeting in December 2003 (Jones et al.): 


(Cp+Ct+Sk+Qt+E2+ Ip) 


ROIS Asset Base 


x100 





The numerator inputs will be weighted according to the relative importance to one 
another and would ultimately be multiplied by the cost of labor hours devoted to that 
category. As described previously, the weighting is very important and was to be 
determined using primarily subjective analysis by the key players. It is important now to 
take a closer look at each of the numerator inputs individually with the realization that the 
formula inputs were still in the development stage. Each of the sub categories would also 


be weighted within each variable using one of the four aforementioned methods. 


Capability (Cp) addresses the integration and testing of the hardware and 
software, the scalability of the suite, and the ease with which the technology can be 
updated. Cost (Ct) accounts for the time versus the resources used, benefits from 
consolidated buys, and lessons learned. Schedule (Sk) assesses the TIF integration 
process, flexibility, and management performance. Quality (Qt) focuses on metrics, 
testing, and interoperability standards. Effectiveness and Efficiency (E2) determines the 


effect of economies of scale, lessons learned, and standard practices. Lastly, Intellectual 
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Properties (Ip) represent the expertise, experience, and training of the personnel involved 
in the process (Jones et al., 2003). 


2. 2" Iteration 


A second iteration of this formula was developed at the May 2004 SPAWAR 
PEO C4I and Space semi-annual meeting. It is shown below including the minor 


modifications that were made to ease the data collection for the formula: 


+ Sk + Ot + Ip) 


NROI(%) =<L. x100 


Asset Base 


The Cost and Effectiveness and Efficiency value added categories were dropped from the 
formula. This was done as a result of a consensus opinion that these two categories were 
largely indistinguishable from elements already present in the remaining four categories. 
The major obstacle still before the effort at this point was developing a data collection 
method that would allow for easy translation of labor hours into the remaining four 


categories. 


Unfortunately, even with the modifications to the baseline formula, the on 
iteration was later determined to be technically unfeasible for two primary reasons. First, 
the value added categories as developed remained too vague. It was concluded that it 
would be too difficult to objectively separate the labor hours from the T-AKE project into 
the individual categories. Also, some of the labor hours might fit into more than one of 
the categories. For example, an hour of labor spent installing equipment may contribute 
to both Capability (Cp) and Quality (Qt). How would these “double value” hours be 
separated to best represent their contribution? Secondly, data collection was not already 
in place to support this formula. Gathering data would have required the creation of new 
data collection methods thought to be too cumbersome and costly. It was also 
determined that it would not be feasible to develop credible values for average labor rates 
for use in the numerator since they vary considerably throughout the organization. 
Consequently, without quality data available to support the Ist or 2" iterations, a change 


of approach was in order. 
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3. 3" Iteration 


The 3rd iteration of the baseline formula was developed at a conference held in 
San Diego at ASC on July 23, 2004. This formula represented a much refined approach 
to the problem relying on a much smaller core group of key players as opposed to the 
more open ended discussions at the semi-annual meetings. In the process of lengthy 
discussion about data availability, validity and applicability with Travis Tillman, the 
Platform Manager for the T-AKE project, the third version of the NROI formula was 


developed by Jones, Bigham and Goudreau as shown below: 


PMx1.1+ENG xl.4+SMx1.15+PRx1.4+ INSTx1.3+SSx1.2+MMx1.1+ Avoidanc€osts- ReworkCosts 
NROI(%)= ( 1)x100 
RadioControlSuiteProgramCosts 





WBS CATEGORIES 

PM — Program Management 

ENG — Engineering 

SM — Supportability Management 

PR — Production (Integration) 

INST — Installation 

SS — Shipyard Support 

MM -— Material Management 
The numerator of the formula was changed considerably as compared with the first two 
iterations. There are four significant differences, the use of SSCC’s existing cost tracking 
method of Work Breakdown Structure (WBS) categories, the utilization of VAFs, the 


addition of avoidance costs, and the capturing of rework costs. 


First, rework costs incurred due to PEO C4I SC actions would be separated out of 
the value added categories. Rework does not add value to the project and, consequently, 
should not enhance the NROI. Second, based on a recommendation that arose during a 
phone conference with Dr. Patricia Phillips, the president and CEO of the ROI Institute, a 
cost avoidance category was added to the numerator (2004). The purpose of this 
adjustment is to capture any costs avoided as a consequence of the role PEO C4I SC 
plays in coordinating the expenditure of DOD resources. For example, if PEO C4I SC 
makes a bulk buy of radio room equipment to support multiple DOD contracts and saves 


a particular sum of money, this would be considered a cost avoided. If a piece of this 
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equipment is then used in the T-AKE ship project, some of this avoided cost should be 


credited to the project as an avoidance cost and improve the project’s NROI. 


Lastly, the WBS categories parallel data already kept by the T-AKE’s platform 
manager with only minor modifications. It was determined that the same process of 
applying weighting factors could also be used to determine VAFs to modify the seven 
existing WBS categories in order to represent the value added by PEO C4I SC to the RCS 
for the T-AKE project. For example, engineering’s VAF of 1.4 indicates that every 
dollar spent on engineering gives the final product a total value of $1.40. A detailed 
description of the WBS categories and VAFs follows. 

4. WBS Categories and Associated VAFs 


The WBS categories used in the numerator of the NROI formula reflect the seven 
categories commonly used across all SSCC RCS Programs. The numerical values for the 
VAFs for each of the WBS categories were initially proposed by Travis Tillman at the 
July ASC conference. However, in an effort to strengthen the accuracy, validity and 
credibility of the weighting methodology, other experts within SSCC were asked for their 
valuation of the different WBS cost categories via email correspondence at the beginning 
of August. The following five members, with their associated titles, then participated 
with the authors in a video teleconference (VTC) meeting on Aug 6, 2004 during which 
final agreement on the VAFs was established: 


Mr. Travis Tillman — T-AKE 1 Class Platform Manager 

Mr. Ken Ayers — LPD-17 Radio Communications System Program Manager, 

Mr. Jim Burgess - CVN-68 Radio Communications System Program Manager 

Mr. Mike Cullison — LHD-8 Radio Communications System Program Manager 

Mr. Kevin Gerald — CVN-68 Radio Communications System Project Engineer 
The VAFs for each of the WBS elements were developed by consensus among this group 
of SSCC civil servants who collectively have over 60 years of experience designing, 


engineering, installing, integrating, testing, and supporting radio communications systems 


for Ship’s Construction, Navy (SCN) platforms. Experience of these members includes a 
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number of different delivery approaches across several platforms spanning the last 20 


years. 


The following discussion of the different WBS categories and their associated 
VAFs is derived primarily from explanations provided by Travis Tillman in person, via 
VTC and through email (2004). Our explanation serves as a summary of comments 
made principally during the VTC where the VAF values were decided upon by the 


participating group members. 


Program Management (PM): A weighting factor of 1.2 was applied to program 
management based on the agreed amount of value added to dollars spent within this 
category. RCS managers have the required education and experience to make decisions 
within their programs that are based on best engineering practice and past experience. 
The RCS managers also have the benefit of discussing and sharing issues with each other 
at SSCC. Currently, the four SSCC RCS managers have, on average, 15 years of direct 
SCN integration and installation experience. SSCC managers are also using a number of 
project management tools. For example, the LPD-17 Class program is using the 
Capability Maturity Model® Integration (CMMI") project. CMMI" is a collaborative 
effort to provide models for achieving product and process improvement. The primary 
focus of the project is to build tools to support improvement of processes used in the 
development and sustenance of systems and products. The output of the CMMI" project 
is a suite of products, which provides an integrated approach across the enterprise for 
improving processes, while reducing the redundancy, complexity, and cost resulting from 


the use of separate and multiple capability maturity models (CMM®s). 


Engineering (ENG): A weighting factor of 1.4 was applied to engineering based 
on the agreed amount of value added to dollars spent within this category. The weighting 
factor of 1.4 was applied to engineering based on the RCS managers’ past experience 
developing engineering products for a number of SSCC delivered radio control systems 
for platforms currently in construction as well as those recently delivered to the fleet 
(Current: CVN 70/77, LHD 8, LPD 17 — 21, T-AKE 1 — 4; Recent: CVN 69/76). The 
core team of engineers has collectively worked on over 15 RCS programs during the past 


20 years. SSCC uses a core team of engineers, leverages engineering products from one 
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hull to the next, uses best engineering practices, to include baseline management, and 


uses an extensive number of Standard Operating Procedures (SOP) to improve efficiency. 


SSCC has also developed engineering products that are tailored to the specific 
platform’s requirements. For example, once a specific drawing is developed for a given 
communication system, such as High Frequency (HF), it becomes the starting point for 
follow-on platforms. Drawings can be sized or scaled to meet follow-on platform 
requirements. The ability to leverage drawings from one platform to the next is one way 
SSCC achieves a significant cost savings in engineering. SOPs are also used as a tool for 
SSCC to minimize cost. SOPs are constantly refined to improve the quality of 
engineering products developed. One of the most beneficial practices employed by SSCC 
is Baseline Management. Baseline Management allows multiple agencies, as well as 
multiple contractors, to stay on the same page. Although engineering products and 
associated development are leveraged across multiple hulls at SSCC, the final products 
are still tailored to the specific platform and allow for the resultant Information 
Capabilities Document, cable run sheets, and “As Built” drawings to be a comprehensive 
set of documentation to be used by both ship’s force and future installers. Because these 
final tailored packages are designed and reviewed before and during the Production phase 
of acquisition, the latest available technology insertion can be employed to provide the 


fleet with the most recent equipment the ship’s C4I baseline will allow. 


Material Management (MM): A weighting factor of 1.1 was applied to material 
management based on the agreed amount of value added to dollars spent within this 
category. Although most of the efforts associated with material management will occur 
regardless of the acquisition strategy or approach, the SSCC SCN Material Management 
team has many years of experience in this category and has developed and consistently 
refined efficient standard operating procedures. The labor for the team and lessons 


learned are shared across multiple platforms constantly improving work practices. 


Supportability Management (SM): A weighting factor of 1.15 was applied to 
supportability management based on the agreed amount of value added to dollars spent 
within this category. The SSCC has a core team of experts that have been supporting the 
SCN process locally for the past eight years. Most of the efforts associated with 
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supportability management are common across all platforms and will occur regardless of 
acquisition strategy or approach. However, the crew familiarization and limited 
maintenance training are developed for the specific platform. The development of the 
supporting documents and classroom packages are leveraged from previous performed 
training. This efficient process provides a cost savings over developing these training 
programs from the ground up each time. SSCC has established a Supportability 
Integrated Product Team (SIPT) that crosses all past, current, and future platforms. The 
SIPT’s sharing of lessons learned across multiple platforms is also incorporated into the 


weighting factor. 


Production/Integration (PR): A weighting factor of 1.4 was applied to 
production/integration based on the agreed amount of value added to dollars spent within 
this category. There is significant “value added” in the integration and testing that SSCC 
performs in each TIF. The production efforts SSCC performs serve several functions. 
They allow for integration and testing of radio room components to occur prior to 
installation on the ship. SSCC production efforts include providing the cables, 
connectors, back shells and other systems not sponsored by the hardware providers, but 
that are required to make individual RF/switching systems into an integrated RCS 
system. The test and integration process is used to allow individual systems to be pre- 
assembled and tested in a lab environment vice onboard ship, where there is potential for 


damage to occur due to the industrial work ongoing during new ship construction. 


The TIF provides the first opportunity to test systems for interoperability with 
other new and legacy systems. The TIF environment also allows for a full “mock up” of 
the shipboard spaces that is used to arrange equipment in an effective and efficient 
environment as well as ensure Human Machine Interfaces (HMI) are located in the 
optimal location for monitoring the equipment. This is essential to ensure that actual 
onboard operators of the equipment can adequately monitor operations within the 
proposed configuration of the radio room. Additionally, the TIF “mock up” ensures the 


required access space for maintainers and operators is maintained. 


Installation (INST): A weighting factor of 1.3 was applied to installation based 


on the agreed amount of value added to dollars spent within this category. This was 
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derived from SSCC’s vast experience and lessons learned from previous SCN RCS 
installations. Many of the efforts associated with the installation process are common 
across different acquisition strategies and approaches (ie. government vs. shipbuilder 
installation). However, the experience of SSCC personnel and associated contractors 
greatly decrease the installation and test time needed during the construction window. 
Also, the installing team is typically much more familiar with the systems to be installed 
and tested. The integration and testing that occurs at each SSCC TIF that was previously 
discussed, greatly reduces the install time as well as the troubleshooting required within 
the industrial environment at the shipyard. Using the full Turnkey Approach for 
installation allows for radio control room installation later in the ship production 
schedule. This allows PEO C4I SC to deliver an RCS that is closer to “state of the art” 


when compared to other approaches, at platform delivery. 


Shipyard Support (SS): A weighting factor of 1.2 was applied to shipyard 
support based on the agreed amount of value added to dollars spent within this category. 
SSCC places civil service and/or contractor personnel on-site at the shipyards that are 
skillfully versed in the various Turnkey processes. They are in place to interface daily 
with the shipbuilder directly. The on-site personnel improve communication between 
PEO C4I SC and the shipbuilder, which allows for quicker identification and resolution 
of issues and concerns. Major issues can lead to stop work orders, which can be 
extremely expensive. The use of shipyard support personnel helps significantly in 
avoiding such work stoppages. 


5. 4" and Final Iteration 


The fourth and final iteration of what had now become the Jones, Bigham, and 
Goudreau NROI formula to be used for project analysis was determined during a VTC 


with Travis Tillman on October 1, 2004. The final version is shown below: 


(WBS- Rework) x VAF+ Avoidance Costs 
Radio Control Suite Program Costs 





NROI% = X 100 


3] 


NROI (%) = 


Shown next is the final iteration with the WBS categories broken out. Note that in the 
interest of space, the WBS categories are presented in their net benefit format with 


rework already removed: 


PM x0.1+ ENGx 0.4+5SM x 0.15 + PR x 0.4+ INST x 0.3 +SS x 0.2 + MM x 0.1 + Avoidance Costs 





Radio Control Suite Program Costs 


There are two notable differences from the third iteration. First the VAFs are 
shown in their net benefit form. Consequently, one is no longer subtracted from the 
fractional portion of the formula. This allows the numerator of the NROI formula to 
represent the net benefit to the project similar to the use of net income in a traditional 
private sector ROI formulation. Secondly, rework costs are no longer subtracted 
separately in the numerator. Instead, rework costs are subtracted directly within their 
respective categories. This gives absolutely no benefit to rework costs determined to be 
required as a result of PEO C4I SC. In the third iteration, rework costs were deducted 
separately and may have inadvertently improved the formula result. Although they were 
also subtracted out in the numerator, this was only after being multiplied by their 
respective VAF. Thus, $10 of rework engineering would have been multiplied by 1.4 
first and then $10 subtracted from that product. This would have given a credit of $4 to 
the project when in fact the rework costs should provide no enhancement to the PEO C4I 


SC NROL 


The fourth iteration of the formula to measure PEO C4I SC NROI for the T-AKE 
project best meets the nine criteria previously discussed. First, the model developed for 
the NROI is simple and theoretically sound. It is free of any complex mathematical 
equations and mimics a commonly used private sector return on asset approach. More 
importantly, it is intuitive such that with brief introduction one can understand how it 


might represent a ROI for dollars invested in the T-AKE project. 


The fourth iteration is also easy to implement. The data required to consider all 


program costs are already kept by platform managers at PEO C4I SC. There is some 
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x 100 


manipulation required to determine rework dollars spent, avoidance costs, overhead 
costs, and to pull travel costs from each of their respective categories. This is done since 
dollars spent on travel costs add no value to the T-AKE RCS, whether the travel is 
necessary or not. However, the additional time required by the platform manager to 


produce the data required is still relatively minimal. 


The final NROI formula meets the credibility standards outlined earlier and 
results are also able to be replicated. The cost data used in the seven WBS categories are 
objective numbers taken directly from the platform manager’s cost reports. This 
maximum use of objective data ensures NROI output can be replicated. The VAFs do 
add subjectivity to the formula, but efforts were taken to minimize this impact. Platform 
managers from four different ship classes worked together, reaching consensus on the 
actual values used for the VAFs in each of the seven categories. In this way subjectivity 
was minimized. Furthermore, this also enhances credibility since personnel that will be 


tasked with the formula’s implementation were included in its development. 


This formula provides the utility to measure NROI for past, present, and future 
project evaluation as well as cross-class comparisons of similar scope. The fourth 
iteration enables RCS NROI calculation across four ship classes since the same VAFs 


remain applicable. This improves formula flexibility. 


The fourth iteration is also comprehensive in nature and considers all of the 
various program costs. The WBS breaks down all dollars spent on the RCS into 
individual categories. All of these are considered when determining value added to the 
RCS. All relevant program costs have also been considered, to include non-value added 


costs such as the service center fee, travel, material, and rework. 


Finally, careful consideration has been given to each of the elements included in 
both the numerator and denominator in the fourth iteration. Specifically, their individual 
and combined effects on the overall NROI output have been verified to be in accordance 
with expectations. For example, additional non-value added costs to a program included 
in the formula have been verified to actually reduce the NROI. This meets the validity 


requirement discussed earlier and will be demonstrated in the following pages. 
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V. FINAL NOTIONAL ROI FORMULA 


A. T-AKE 1 NROI 


The final NROI formula was developed to be applied practically and represents 
the most appropriate ROI developed for and adapted to the design, equipment 
procurement, testing, installation, personnel training, and certification of the radio room 
equipment and associated gear for the first ship of T-AKE new construction. The 
calculation of the T-AKE 1 RCS NROI as of August 27, 2004 and the projected overall 
NROLI are provided and analyzed below. 


Actual costs collected through August 27, 2004 and projected totals are shown in 
Table 7 below for each of the seven WBS categories. Note that nothing has been spent as 
of August 27th, on either Installation or Shipyard Support since none has been required 


up to this point in the program. 





























Table 7. T-AKE 1 WBS Category Cost Data 
WBS Category and VAF ea aaa Projected Totals 
August 27, 2004 
Program Management (0.2) $549,432 $765,932 
Engineering (0.4) $1,411,618 $1,451,618 
Support Management (0.15) $157,897 $242,897 
Production (0.4) $469,349 $869,349 
Installation (0.3) $0 $650,000 
Shipyard Support (0.2) $0 $75,000 
Material Management (0.1) $101,899 $191,899 











Table 8 shows actual costs through August 27, 2004 and projected totals for the 


T-AKE RCS project for non-value added costs associated with the program. The service 
4] 





center fee represents the share of corporate costs or overhead charged to all SPAWAR 
projects. Travel costs are normally charged to the individual WBS categories. However, 
they are separated from their respective WBS categories since they are a non-value added 
cost to the T-AKE 1 ship. The role of PEO C4I SC in material procurement is like that of 
a middle man, buying existing equipment from suppliers and delivering them to the 
buyer. Therefore, the cost of the material itself is considered a non-value added cost. 


PEO C4I SC adds value to materials only after installation, testing and integration in the 




















radio room. 
Table 8. T-AKE 1 Non-value Added Actual Costs and Projected Totals 
Actual Costs through 
Non-Value Added Cost Projected Totals 
August 27, 2004 
Travel $121,119 $181,119 
Service Center Fee $39,204 $64,204 
Material $1,487,840 $1,912,840 
Engineering Rework (must be subtracted from 
ENG category prior to VAF modification) $5,600 $5,600 














Due to lack of specific data collection, PEO C4I SC generated rework was 
estimated to be approximately $5,600 in the engineering category. Also, no avoidance 
costs were tracked for the T-AKE 1 RCS project. They are assumed to be zero in an 
effort to be conservative. As discussed earlier, rework is simply removed from the WBS 
category prior to modification by the VAF; it is included in the program costs. All the 
data have now been presented to allow calculation of an NROI for the T-AKE 1 RCS 
project as of August 27, 2004 and an NROI based on projected totals. These calculations 
are shown below. The actual NROI as of August 27, 2004 is calculated as follows: 
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(WBS- Rework) x VAF+ Avoidance Costs 
T - AKE1 RCS Program Costs 


NROI% = X 100 





_ PMx0.2+(ENG—Rework) x 0.4+SMx0.15+PRx0.4+ INSTx0.3+SSx 0.2 +MMx0.1+ AvoidanceCosts 
7 T - AKE1 RCS ProgramCosts 


X100 


_ $893,907 +0 


= X100= 20.6% 
$4,338,358 


The projected NROI at completion of the T-AKE 1 RCS program is calculated as 


follows: 


_ $1,344,957 +0 
$6,404,958 


X100=21.0% 


More detailed calculations are provided in Table 9 on the next page. Notice that the 
actual and projected NROI remains relatively stable. Of course this assumes that no 
more PEO C4I SC generated rework will be incurred and avoidance costs will continue to 
be zero. The slight differences in NROI are due to a shifting in proportions between the 


WBS categories as the project matures. 
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T-AKE 1 Cost Schedule and NROI Determination 








Table 9. 


B. NROI COMPARABILITY FOR ONE SHIP CLASS 


The developed NROI formula possesses the flexibility to enable calculation of the 
NROI at various times for any given T-AKE ship RCS project. An NROI could be 
estimated prior to project initiation based on projected WBS and associated non-value 
added costs. It can be calculated at any time during the project cycle with actual costs up 
to a given date. NROI is also able to be calculated at project completion. If senior level 
buy in for the proposed formula was obtained, this could be used as one of the 
benchmarks to indicate the amount of output PEO C4I SC is providing for the program 
dollars invested. A priority of the CNO discussed earlier, this would begin to answer the 


request of linking actual outputs to dollar investments. 


Comparisons made of the T-AKE RCS project NROI at different times during the 
acquisition cycle must be carefully considered. The underlying assumptions and 
circumstances must be evaluated cautiously since several factors can attribute to NROI 
increasing or decreasing at discrete points in the project cycle. These factors include 
changes in the amount of PEO C4I SC generated rework, a shift in the proportional 
amounts of value added costs in the WBS categories, adjustments made in avoidance 


costs, or changes in the amount of non-value added costs attributed to the project. 


PEO C4I SC generated rework costs are subtracted from the numerator of the 
NROI formula since they provide no net benefit. They are included as a program cost of 
the project in the denominator. Therefore, as the amount of PEO C4I SC generated 
rework increases over the project cycle, a negative impact on the NROI calculated will 


occur. This is one possible explanation for a decreasing NROI over the life of a project. 


Shifts in the proportional amounts of the value added costs in the WBS categories 
over the life of the project will also affect the NROI. This may be due to no fault of the 
program itself. For example, one of the highest value added categories is engineering. 
Table 7 from the previous section illustrates how as of August 27, 2004 most of the 
engineering dollars were spent while some lower value added categories such as shipyard 
support and installation remained unspent. This shift in WBS category costs is a natural 
result of the T-AKE project progression. Significant engineering dollars are spent up 


front to design and develop the RCS while installation takes place much later in the cycle 
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after most engineering dollars have been spent. This shifting of WBS category costs 
could be responsible for changing the NROI between two discrete points in the project 
cycle since each WBS category is converted to value added using numerically different 
factors. A non-proportional shift in WBS costs to lower value added categories will 
lower the NROI. Conversely, a non-proportional shift in WBS costs to higher value 
added categories will have the effect of increasing the project NROI. 


Costs avoided by PEO C4I SC during the life of the T-AKE project will have a 
positive effect on NROI. Avoidance costs are added in their entirety to the numerator 
and are not included in the denominator. This is also a possible explanation for 
increasing NROI over the life of the project. Unfortunately, no avoided costs were 


captured by the current method of data collection used for this project. 


Varying amounts of non-value added costs over the course of the T-AKE project 
will have a direct effect on the project’s NROI. Non-value added costs include expenses 
such as travel, overhead, and purchased material. These costs are part of the NROI 
formula’s denominator but are not included in the numerator since they provide no “PEO 
C4I SC added value” to the radio room. Therefore, changing amounts of non-value 


added costs will impact the NROI between any two points in the project cycle. 


NROI between different ships of the T-AKE class can also be evaluated. For 
example, the NROI of the T-AKE 1 RCS can be compared to the NROI of the successive 
T-AKE ship radio rooms as the actual or projected cost data become available. This 
could potentially indicate the ROI for employing PEO C4I SC for multiple ships within 
the T-AKE ship class or indicate problems with a specific T-AKE hull. As with the 
comparisons made at different points in one T-AKE RCS project cycle, comparisons 
made between successive T-AKE radio rooms must also be evaluated carefully. The 
potential causes for changing NROI analyzed and described previously for a single T- 
AKE project all apply to comparisons made between different T-AKE hulls. However, 
additional considerations must be made when making comparisons between hulls. The 
natural learning curve between successive identical platforms and the significant potential 


for increased avoidance costs should also be assessed. 
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A learning curve naturally occurs when repetitive tasks are performed by an 
organization such as PEO C4I SC. This will occur for many reasons including increased 
familiarization with the tasks required and improved project coordination. The DoD not 
only considers this, but expects it as well, when awarding contracts for multiple item 
procurements such as successive ship hulls (Rendon, 2004). Improved efficiencies will 
be realized and some costs from the first RCS will be avoided on subsequent platforms. 
For example, much of the engineering design and development costs will be avoided on 
later T-AKE hulls since they will be “cookie cutter” examples of the first. Also, it is 
likely that SPAWAR generated rework on a previous T-AKE hull will be avoided on 
later T-AKE hulls due to lessons learned. This improved efficiency and avoided costs 
can affect the NROI either positively or negatively. Increased avoidance costs for 
successive hulls would increase the NROI from one hull to the next. Conversely, 
significantly reducing engineering costs in later hulls may have a negative effect on 


NROI since this is one of the two highest value added WBS categories. 


It is essential to understand that any ROI formulation, in and of itself, has very 
little meaning. It requires a comparison to some relatively similar baseline or benchmark 
to gain significance. It is also important that when making these comparisons, the 
context of the two numbers being compared is well understood. This is no different than 
when considering traditional ROI. Consider an investor who gained five percent on his 
or her portfolio over the previous year. For this to be considered a respectable ROI 
would depend on several factors. If this was a low risk investment, five percent may be 
adequate. If this were an aggressive investment, it is likely that five percent would be 
considered sub par. However, if the economy was in a recession, five percent return may 
be superior regardless of the investment profile. The point is that an NROI requires a 
comparison data set to give it meaning. Thus, careful consideration must be given to the 
context of two NROI being compared to fully understand their significance, just as with 


comparing traditional ROI in the private sector. 
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C. LPD-19 NROI 





Figure 2. LPD-17 Class (Global Security, 2004) 


LPD-19 is the third vessel of the new LPD-17 ship class. Data were collected in 
the same WBS category format to allow calculation of an NROI in a similar manner that 
the T-AKE NROI was ascertained. Recall that the same VAFs used for the T-AKE RCS 
are also valid for the LPD-17 class. The details of the NROI calculations for LPD-19 are 
based on actual cost data through September 4, 2004 and on projected totals. WBS 


category costs are shown in Table 10. 


























Table 10. LPD-19 WBS Category Cost Data 
WBS Category and VAF dca a aaa Projected Totals 
September 4, 2004 

Program Management (0.2) $1,352,275 $1,985,275 
Engineering (0.4) $195,000 $612,000 
Support Management (0.15) $1,096,160 $1,328,260 
Production (0.4) $3,128,951 $6,252,951 
Installation (0.3) $1,293,766 $3,086,509 
Shipyard Support (0.2) $0 $0 
Material Management (0.1) $1,186,544 $2,373,088 
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The value added costs represent those spent for the RCS of the LPD-19 to allow 
better comparability to the T-AKE results. Non-value added costs as of September 4, 
2004 and projected totals are shown in Table 11. Note that to date no SPAWAR 
generated rework has been incurred for LPD-19. This is most likely due to the fact that 
she is the third ship of her class and many lessons have already been learned whereas the 
previous data were for the inaugural ship of the T-AKE class. In an effort to remain 


conservative, avoidance costs for LPD-19 are also assumed to be zero since they were not 

















tracked. 
Table 11. LPD-19 Non-value Added Actual Costs and Projected Totals 
Actual Costs through 
Non-Value Added Cost Projected Totals 
September 4, 2004 
Travel $71,103 $229,353 
Service Center Fee $85,433 $117,083 
Material $2,059,481 $2,271,481 
Engineering Rework $0 $0 

















The actual NROI for the LPD-19 as of September 4, 2004 is calculated as 


follows: 


(WBS- Rework) x VAF+ Avoidance Costs 
LPD-19 RCS Program Costs 


NROI% = X 100 





_ PM x0.2 +ENGx 0.4+SM x 0.15 + PR x 0.4 + INST x 0.3 +SS x 0.2 +MM x 0.1 + Avoid Costs 
LPD -19 RCS Program Costs 


X 100 





_ $2,271,244 +0 
$10,468,713 


X100= 21.7 % 
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The projected NROI at completion of the LPD-19 RCS program is calculated similarly as 


follows: 


_ $4,503,296 +0 


X100 = 24.67 % 
$18,256,000 





The NROI of LPD-19 does increase over the life of the project. This is primarily due to 
the significant number of dollars still left to be spent on the high value-added categories 
of production and installation. More detailed calculations are provided in Table 12 on the 


next page 
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LPD-19 Cost Schedule and NROI Determination 


Table 12. 
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D. NROI COMPARABILITY FOR MULTIPLE SHIP CLASSES 


NROI comparisons between PEO C4I SC installed radio control rooms can also 
be made between ship classes. Some of the same factors that must be considered when 
making comparisons between radio rooms of the same hull must also be considered here. 
Differences in avoidance costs realized, the proportions of costs within the WBS cost 
categories, and the amount of PEO C4I SC generated rework and other non-value added 
costs will all promote differences in the two NROI. However, there is the potential for 
these factors to have a much greater effect when comparing NROI of radio rooms 
between different ship classes. Differences in the learning curve, the relative amounts of 
value added costs and non-value added costs, and the VAFs themselves must all be 


considered when making comparisons between different ship classes. 


The learning curve will vary significantly between hulls from different ship 
classes for several reasons. First, technical complexity between the radio control rooms 
of different ship classes can vary widely. The RCS installed in a T-AKE class vessel 
would not be considered very complex when compared to the RCS of an aircraft carrier. 
Increased complexity can either increase or decrease the learning curve. Also, the 
number of hulls being built within a given ship class will affect the amount of benefit 
realized from a learning curve. If only a few hulls are to be constructed as compared to a 
ship class with a large number of hulls designated, there will be less benefit from the 
learning curve achieved for the smaller ship class. The different learning curves may 


affect the amount of cost avoidance realized from hull to hull and, thus affect the NROI. 


The relative amounts of value added costs between the WBS categories will also 
produce differences in NROI compared between ship classes. A warship on the cutting 
edge of technology would require larger amounts of dollars to be spent in the 
engineering, production (integration), and installation categories than an auxiliary ship 
with average radio control room technology. These shifts affect the net benefit, or 
numerator, determined in the NROI formula. Consequently, shifts in WBS costs between 


ship classes will affect the resulting NROI. 


The amount of non-value added costs may vary considerably between hulls of 


different ship classes. The amount of material required for purchase to support the radio 
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room, the amount of travel required, and the service center fee charged will all affect the 
NROI. Most notably is the amount of material that needs to be acquired. The CVN-68 
class will require appreciably more material to be purchased than an auxiliary ship like 
the T-AKE class. This large difference in non-value added costs due to material 
differences will dampen the NROI for the carrier since the value of the denominator will 
increase with no increase in the numerator. The amount of travel required may also vary 
considerably depending on the location of the shipyard chosen to build a ship class. 
Also, depending on the allocation method for overhead, the amount of the service center 
fee may differ considerably between ship classes. The different relative amounts of non- 
value costs may have a large impact on the NROI when comparing hulls from different 


ship classes. 


Finally, the VAFs used to determine an NROI for a given ship class can also 
differ. For the context of this thesis, the VAFs chosen were developed by SSCC resident 
experts from the T-AKE, CVN-68, LPD-17, and LHD-8 classes of ships. These VAFs 
represent their best judgment as to the value PEO C4I SC adds to dollars spent within the 
respective WBS categories. However, the VAFs would likely not be the same for a 
Virginia class submarine. It is relatively easy to surmise that such differences would 


change the bounds of the equation entirely. 


NROI comparisons made between hulls of different ship classes must be made 
cautiously. The learning curve, relative amounts of value added costs and non-value 
added costs, and the NROI VAFs can all differ appreciably. This may skew the results 
to indicate one project is operating better than the other when in fact some of the 
differences may be attributable to the uniqueness of the project itself. Let us consider 
again comparisons of traditional ROI measurements. The return in the bond market is 
not expected to equal returns in the stock market. They are two different investment 
vehicles. However, the expected return on bonds can be approximated if the return on 
the stock market is known once an adjustment is made to take into account the different 
investment options. This adjustment, commonly called a risk premium, is based on 
market history. A similar type of adjustment, once determined, could be used to compare 


NROI of entirely different ship classes. 
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E. COMPARISON OF RCS NROI: T-AKE 1 AND LPD-19 


The LPD-19 and T-AKE 1 RCS NROI both surpass twenty percent. This 
indicates over a twenty percent ROI based on the NROI formula developed. On the 
surface, both projects appear to add significant value to their respective radio rooms. 
However, it is important to consider the frame of reference used when making this 
judgment. A twenty percent return in the stock market by most benchmarks is excellent. 
Conversely, to more accurately evaluate the value added to a radio room, a new 


benchmark must be created based on similar data collected over time. 


The LPD-19 RCS project has both a slightly higher NROI to date and at project 
completion based on projected totals. Taken at face value, this would indicate that the 
LPD-19 project funds are better spent since dollar for dollar they add more value than the 
funds spent on the T-AKE 1 venture. However, as indicated previously, NROI results 
cannot be compared without taking into consideration other factors. One must consider 
the complex underlying circumstances for derivation of the numbers to make a fair 
comparison. Best use of the NROI is between ships of the same class vice between 
classes. However, additional refinement of the methodology may make such comparison 
between platforms more valid. We remind readers that ROI in itself is but one piece 
among many types of financial information available to platform, program and financial 


management decision makers when comparing actual and potential investment options. 


The proportions of dollars spent within the WBS categories for the two ships vary 
considerably. Table 13 on the following page shows a breakdown of each individual 
WBS category cost as percentage of total WBS dollars projected to be spent. Most of 
these differences are inherent to the project. The largest difference is in the engineering 
WBS category. A much larger proportion of WBS dollars are spent on engineering for 
the T-AKE 1 RCS than on LPD-19. This is intuitive since significantly more engineering 
dollars are required for the first ship of any class. This is the case with the T-AKE-1 
RCS. LPD-19 is the third ship of the LPD-19 class, and consequently, requires less 


expenditure in the engineering WBS category. 
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Table 13. Individual WBS Totals as a Percentage of Total Projected WBS Costs 





























WBS Category T-AKE 1 RCS (%) LPD-19 RCS (%) 
Program Management 18 13 
Engineering 34 4 
Support Management 6 8 
Production 20 40 
Installation 15 20 
Shipyard Support 2 0 
Material Management 5 15 











The LPD-19 RCS requires considerably more dollars to be spent in the production 
WBS category. This includes both absolute dollars and as a percentage of total WBS 
category costs. This also is to be expected since the LPD-19 RCS project is 
approximately three times the size of the T-AKE 1 RCS project based on total program 
dollars. The LPD-19 RCS is more complex in nature than the RCS of the T-AKE 1 
vessel. Production is a high value added WBS category and this difference is the leading 
contributor to LPD-19’s higher NROI. Thus, while LPD-19 does have a slightly higher 
NROL, this is due largely to the inherent differences between the programs. 


LPD-19 spends no dollars in the shipyard support category. This is not a large 
contributor to the NROI differences. However, it is important to re-emphasize that when 
comparing ships from different classes some of the VAFs chosen may be different or not 
apply at all, as in this case. This must be carefully considered when making these 


evaluations. 


The comparison of non-value added costs is shown in Table 14. Note that in all 
cases except for rework, LPD-19 has a larger burden of non-value added costs. This is 


expected for both the material required and the service center fee considering the larger 
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scope of the project. Travel is a separate function of the chosen location to build the ship. 


LPD-19 should also be expected to have less rework costs given that it is the third ship of 


the LPD-17 class. 

















Table 14. Comparison of Total Projected Non-value Added Costs 
Non-value Added Cost T-AKE 1 RCS ($) LPD-19 RCS ($) 
Material 1,912,840 2,271,481 
Service Center Fee 64,204 117,083 
Travel 181,119 229,353 
Estimated Rework 5,600 0 














One aspect not indicated in the numbers is that an integral system required for 
each RCS is procured in an entirely different manner. The HF system for the T-AKE 1, 
due to its uniqueness, was procured by the RCS manager using program dollars allocated 
to the T-AKE 1 RCS program. However, the HF system for the LPD-19 was procured 
using dollars from a separate pot of money other than from the LPD-19 RCS program. 
The different methods of procurement and associated accounting for program dollars do 
affect the projects NROI as the formula stands. Since material is treated as a non-value 
added cost, increases in dollars spent on material will lower a project’s NROI. Therefore, 
the effect is to lower the T-AKE’s NROI and, subsequently, increase the LPD-19’s 
NROI. Given that the HF system can cost in upwards of millions of dollars, the effect on 


these NROI is not trivial. For better comparison, this adjustment should be made. 


Finally, while avoidance costs were not tracked due to the timing of this 
endeavor, they could have potentially had a major impact on the NROI of the two 
projects. For example, since LPD-19 is not the inaugural vessel, it is plausible to 
presume that many costs were avoided on the third ship versus those that were incurred 


on the first. Also, since it is likely that large equipment purchases for multiple class ships 
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result in a cost savings, these would also increase the avoidance costs credited to a 


particular program. These avoidance costs would positively impact the project NROI. 


Considering the number of inherent differences between the programs and the 
small numerical differences between the NROI calculated, it cannot be concluded that the 
differences reflected in the NROI calculation are due to one program operating more 
efficiently than the other. Unfortunately, at this point evenhanded comparisons made of 
NROI between ship classes can be difficult, depending on the degree of scope differences 
between the programs. While the LPD-19 and T-AKE 1 RCSs both use the same VAFs, 
many disparities still exist between the programs that must be considered to level the 
playing field between the projects. This comparison might further be complicated if the 
use of different VAFs were required. Continued data collection and NROI calculations 
need to be made in order to better establish an expected or normal benchmark to facilitate 
comparison and more accurately understand how differences in various programs can be 
expected to affect the NROI. Armed with this knowledge, comparisons between ship 


classes can be more easily made. 


F. CHALLENGES 


1. Formula Limitations 


There are several challenges ahead for PEO C4I SC to successfully implement the 
developed NROI process and formula application within the organization. First, the 
subjective nature of the VAFs will require senior level management buy-in to strengthen 
the formula’s credibility. Phillips and Phillips point out three audiences which must buy 
in to the ROI process for it to be useful. They include the practitioners who are 
responsible for implementing the formula and held accountable by the results, senior 
level management who hold the practitioners accountable for these results, and 
researchers (Phillips and Phillips, 2002: 12). The VAFs and the entire NROI formula 
were developed in cooperation with several experts at the practitioner level. However, in 
order for the NROI formula to satisfactorily represent program—performance, senior 
managers responsible for evaluating this performance must buy in to the formula and, 


more importantly, the selected values of the VAFs. For example, senior management 


a) 


must agree that an engineering dollar spent on the T-AKE platform RCS produces 
approximately one dollar and forty cents worth of value. The NROI formula will not be 
used by the practitioners if it will not hold weight with their managers. This is true of 


any new process initiatives in the work place. 


Segregation of responsibilities with regard to data collection and evaluation must 
also be achieved to strengthen the credibility of the results. In our case study due to 
personnel availability, Travis Tillman provided assistance in both formula development 
and data collection. However, as Platform Manager he is also responsible for the results 
of the T-AKE Class RCS. There is a clear motive to skew the results if the person 
responsible for the results also collects and interprets the data. Segregation of these 
duties is necessary going forward if the NROI metric is to be utilized within PEO C4I SC 


for actual management decision making. 


Increased resources will be required from PEO C4I SC to modify the NROI 
development process and collect data for the evaluation of other projects. No two 
projects within the PEO C4I SC organization are exactly alike and some modification of 
the work done for this project will be required. This will demand managerial effort to 
develop an approved NROI measurement and some amount of resources to collect the 


required data for other projects. 


Ideally, the NROI process should be implemented at the inception stage of a 
project instead of during or after the project is in progress. The T-AKE RCS program 
was already well underway before this research began. Consequently, some data 
collection opportunities were missed and some data either had to be estimated or 
discarded altogether in an effort to remain conservative. This limits the accuracy of the 
formula. For example, rework costs were not collected up to the point the formula was 
created and had to be estimated. Avoidance costs were also not being tracked and were 


assumed to be zero since including an artificial number would have increased the NROI. 


The final limitation is that the makeup of the formula itself limits the possible 
range of results more than a traditional ROI formula would. Traditional ROI used in the 
private sector is not as bound by the formula itself. ROI values in the private sector 


normally range from negative one hundred percent (a total loss which could also be 
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surpassed in rare cases) to an unbound positive value. Negative and positive ROI are 
routinely observed in the stock market. Any organization with a negative net income will 
have a negative ROI. Organizations with a positive net income will have a positive ROI. 
However, the NROI formula developed for PEO C4I SC prevents a negative number for 
a result. The lowest value achievable would result from minimum WBS costs, significant 
rework costs, and no avoidance costs. Theoretically, this could be zero, although 
practically it would have to be some number above zero since some amount of work 


performed will not be classified as rework. 


The values chosen for the VAFs also place an upper bound on the value attainable 
for a NROI. They place an explicit upper bound if there are no avoidance costs. 
Consider the VAFs developed for the T-AKE RCS project. The largest values of 0.4 are 
from the WBS categories of engineering and production. The highest NROI theoretically 
achievable without avoidance costs would result from all costs placed into these two 
categories and no non-value added costs like rework, overhead, or travel. This would 
place an upper theoretical limit of the NROI of forty percent. Consequently, the VAFs 
chosen for a particular project will set bounds for the potential results derived from the 
NROI formula. This limitation must be considered when evaluating the results. 


2. Applicability Limitations 


The NROI formula developed for the T-AKE RCS has clear applicability 
limitations to other DoD and public sector programs. This application of the NROI 
formula is, by itself, relevant only to the platforms for which the VAFs were developed. 
These include radio rooms from the T-AKE, LPD-17, CVN-68, and LHD-8 ship classes. 
These factors define how the key players involved in their development numerically 
define how dollars spent within the different WBS categories result in added value to 
their respective RCS projects. These are not applicable to other programs. The VAFs 
would most likely change if they were developed for another public sector organization 


since their personnel may view the process of adding value much differently. 


The use of WBS categories is also not universal across the DoD. WBS categories 
are primarily used within the acquisition community to allow program managers and 


contractors to breakdown large acquisition projects into smaller more manageable tasks. 
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Therefore the use of these WBS categories as a data collection basis for an NROI 


application is limited to this community. 


The NROI development methodology of defining how an organization adds value 
via its products or services and creating a representative metric is not unique to PEO C4I 
SC. The methodology employed in this thesis can be applied to any public sector 
organization that is willing to go through the necessary steps to define how they create 
value in the products or services they provide. Several public sector organizations have 
already successfully implemented ROI metrics as one benchmark to demonstrate their 
performance. However, due to the distinct public sector outputs, this endeavor is often 
times challenging. Private sector organizations typically have the common goal of 
producing profit. Conversely, public sector organizations have a myriad of different 


goals most of which do not include the generation of profit. 
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VI. SUMMARY AND CONCLUSION 


A. SUMMARY 


This thesis has probed the efforts of diverse institutions in the quest for a 
workable method of determining ROI in the public sector. Past efforts provided valuable 
lessons learned which were identified and discussed. The United States Postal Service 
was notably successful, but perhaps only because they function much more like a private 
company than is the norm for public sector entities. In Australia, the bold effort to 
develop a “value added” approach within the entire government budget process made 
some progress, but fell short primarily due to a focus on analysis lacking a quantifiable 
formula for determining the output. The Royal New Zealand Navy enjoyed relative 
success but was hampered in some ways by an inability to screen out other influences 
besides the bonus scheme for retention of MEs. The United States Navy Dental 
community effort was largely successful in the development of an ROI methodology, but 
fell short in their ability to use the output since they lacked trust in the method developed 


for determining some of the inputs. 


The ultimate point of this thesis was to develop an NROI formula specifically 
tailored for the SPAWAR PEO C4I SC RCS. The sequential process of NROI formula 
development for the T-AKE RCS was reproduced as a way of highlighting the significant 
challenges inherent in this effort as well as to serve as a roadmap for others to follow. It 
can only be hoped that use of this roadmap may help others to avoid some of the 
distractions and obstacles that the lessons learned exposed. The utility of the final 
formula was gratifying in that it was readily usable not only for the T-AKE class but also 
as a comparison tool for the RCS projects from LPD-17, CVN-68, and LHD-8 ship 
classes. Of course adjustments must be created to directly compare ships with vastly 


different asset bases in play, but the formulaic approach is sound. 


B. CONCLUSIONS 


The initial goal of this effort was to develop a formula that would ultimately 
provide DoD with a means of incorporating NROI criteria in the decision making process 
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from the highest levels of government. From there the scale was refined to creation of an 
NROI formula specifically tailored for SPAWAR PEO C4I SC leadership to utilize 
concerning new ship construction. An additional reduction in scope was soon 
forthcoming with the new focus zeroed in on the T-AKE 1 RCS. The narrowing of scope 
from first intentions to ultimate conclusion was necessitated by the sheer complexity and 
inherent resistance to this effort. The relative failure to successfully evaluate ROI in the 
public sector despite past efforts is well known and has served to create an indifference to 
trying among many civil servants and military personnel. The result of this effort should 


at least serve to embolden those who would continue to press on in this endeavor. 


Perhaps the most important lesson learned throughout this process was that the 
scope of such an effort must remain focused to be successful. Once NROI formulae are 
established at the project level and successfully demonstrated, an expansion of utilization 
could be envisioned which may ultimately lead to the availability of ROI data for senior 
DoD decision making. The goal now is to continue moving in the right direction to 
answer the CNO’s call to action for developing a method of determining ROI to assist the 
Navy decision making process. The CNO reaffirmed this vision in a visit to the Naval 
Postgraduate School on March 17, 2004. In the words of Douglas A. Brook, Dean of the 
Graduate School of Business and Public Policy at NPS: 

The CNO is clearly focused on the business management aspects of 

running the Navy. He is challenging the Navy to focus on effectiveness 


and costs rather than what he perceives to have been a "readiness at any 
cost" approach (Brook, 2004). 


Dean Brook also pointed out that one of the specific tasks noted by the CNO for the 
faculty and students of NPS included finding “a way to calculate ROI.” It is our sincere 
desire and belief that we have assisted in attaining this goal through the development, test 


application and evaluation of the NROI methodology presented in this thesis. 
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